Digital cameras with a multitude of operational features including but not limited to exposure control, white balance and auto focus, have been a consumer staple for decades. As camera complexity has increased, required actions by the user to operate digital cameras have also increased.
An electronic imaging system depends on a lens system to form an image on an image sensor in order to create an electronic representation of a visual image. Examples of such image sensors include charge coupled device (CCD) image sensors and active pixel sensor (APS) devices (APS devices are often referred to as CMOS sensors because of the ability to fabricate them in a Complementary Metal Oxide Semiconductor process). A sensor includes a two-dimensional array of individual picture element sensors, or pixels. Each pixel is typically provided with either a red, green, or blue filter, as for example described in commonly assigned U.S. Pat. No. 3,971,065, to Bayer, entitled “Color imaging array,” so that a full color image can be produced. Regardless of the type of image sensor employed (e.g., CCD or CMOS) the pixel acts as a “bucket” in which photo-generated charge is accumulated in direct proportion to the amount of light that strikes the pixel during the capture of an image by the electronic imaging system.
The image sensor gathers light for an interval of time called the exposure time or integration time to make a correct exposure during image capture. Based on brightness measurements of the scene to be imaged, the electronic imaging system, typically with an autoexposure system, is employed to determine a suitable exposure time that will yield an image with effective brightness and an effective signal-to-noise ratio. The dimmer the scene, the longer the amount of time the electronic imaging system must use to gather light to make a correct exposure. If motion of objects in the scene relative to the image capture device is present during image capture, motion blur can be present in the captured image. Motion blur is caused when the relative motion between the camera and a point in the scene causes the point in the scene to be imaged over a number of pixels on the imager during the time of exposure. The motion blur during an exposure can be described by a motion blur point spread function, which quantifies the relative exposure of the point in the image onto each pixel during the time of exposure. If there is no motion blur, the motion blur point spread function is a simple spike (i.e., there is no motion blur). As the point moves more during exposure the motion blur point spread function spreads out to cover more pixels.
Motion blur can be caused by camera movement, in which case the motion in the projected image is largely the same throughout the image and can be described by a motion blur point spread function that is substantially constant throughout the image. This is referred to as global motion. It is also common for object(s) within the scene to move independently, such as a person moving within the scene. This is referred to as local motion and produces a different motion blur point spread function for the moving object than for other parts of the scene. Either or both of these types of motion may be present during composition and capture of a scene.
Global motion is normally easier to measure because only a small number of parameters, such as a single motion velocity vector, need to be estimated. Further, inertial devices such as gyroscope(s) or accelerometer(s) can be used to provide data on camera motion.
Both global and local motion are well described by velocity vectors (how quickly a point in the projected image is moving), which can be characterized by a direction and a speed quantified in pixels per second. Both local and global velocity can change with time, and the integral of the velocity over time provides information on the projected point's position over time. Increasing velocity or exposure time will generally increase the motion blur during an exposure. In contexts where it is unimportant whether a motion blur is due to local motion or global motion, the present disclosure uses the term scene motion.
A number of methods to reduce global motion blur are known to those in the field. One method is to use an image stabilization system. Such methods typically use an inertial measurement device (e.g., a gyroscope or an accelerometer) to measure the motion of the image capture device during capture and then use a special lens with a lens element or lens group that can be moved laterally to cause the image formed by the lens on the image sensor to move in a direction that compensates for the image capture device motion. In other embodiments, the image sensor itself can be moved laterally to compensate for the image capture device motion.
A method that can be used to correct for motion during the capture of video image is described in U.S. Patent Application Publication 2006/0274156 to Rabbani et al., “Image sequence stabilization method and camera having dual path image sequence stabilization.” This approach is based on a digital shifting of individual frames in a captured video sequence to compensate for movement of the digital camera. While this method cannot reduce motion blur in a single frame, it is effective to stabilize a sequence of captured video images to reduce the effect of camera shake.
None of the above-described methods are effective to reduce the effects of local motion blur. One method to reduce local motion blur is to shorten the exposure time to a setting which is shorter than the exposure time selected by an autoexposure system that considers only scene brightness. The resulting images will be darker and will have a lower signal-to-noise ratio. An analog or digital gain can then be applied to the pixel values in the image to brighten the darker images, but those skilled in the art will recognize that this will result in noisier images.
U.S. Pat. No. 7,657,164 to Nomura et al., entitled “Subject shake detection device, imaging device, control method thereof, control program, and recording medium,” describe the use of gyros and image analysis to estimate camera shake. The exposure time is adjusted to limit motion blur according to a predefined threshold.
U.S. Pat. No. 7,720,376 to Weinberg et al., entitled “Camera with acceleration sensor,” teaches a camera with an acceleration sensor. A sensed acceleration is used in the process of determining a minimum shutter speed that should be used for a particular focal length.
U.S. Patent Application Publication 2007/0188617 to Stavely, entitled “Apparatus and method for reducing image blur in a digital camera,” teaches determining camera motion information using motion sensors and image analysis. The motion information is used to control the moment of image capture to provide reduced motion blur.
Another method to reduce local motion blur is to gather more light using either a lens with a larger aperture or an image sensor with larger pixels, thereby enabling the use of a shorter exposure time. This approach can produce images with reduced motion blur and acceptable noise levels. However, the current industry trend in electronic imaging systems is to make image capture devices more compact and less expensive. High-grade optical elements with large apertures and image sensors with larger pixels are substantially more expensive, and are therefore not practical for many applications.
Another method to reduce local motion blur is to supplement the available light with a photographic flash in order to reduce the effective exposure time. A photographic flash produces a strong light flux that is sustained for a small fraction of a second. The photographic flash can be an electronic flash with a xenon tube, a light emitting diode (LED) or an array of LEDs, or some other light source controlled or triggered when the camera is capturing an image. The actual exposure time can be set to a short value which is marginally longer than the flash duration. Therefore, the motion blur caused by either global or local motion during the exposure can be significantly reduced. However, flashes are not effective in bright lighting and fast moving objects in bright lighting can still produce local motion blur. In addition, flash photography is typically only useful if the distance between the flash and the scene being photographed is small. Flash photography also tends to produce artifacts such as red eyes and very bright areas or dark areas in the captured image, which many people find objectionable.
Methods that can mitigate local motion blur are generally effective against global motion blur as well, but have the limitations described above.
Conventional solutions for selecting exposure time typically use one or more standardized settings, or respond to operator mode settings to obtain an exposure time. FIG. 1 shows a flow chart of a typical exposure control system 200 for a digital camera performing autoexposure. In assess scene brightness step 210, the camera assesses the scene brightness either with a scene brightness sensor or with an analysis of a preview image. In determine capture mode step 220, a capture mode setting 225 is determined based on the measured scene brightness and any operator-selected user interface settings or standardized settings. In determine exposure index step 230, the exposure index setting 235 (EI) is determined in accordance with the measured scene brightness and the capture mode setting 225. Those skilled in the art will recognize that exposure index is a standard way to quantify the amount of light necessary for a good exposure. For film-based cameras, the exposure index is usually set based on the film speed, or ISO rating, which is related to the film sensitometry. Conversely, in digital cameras, the exposure index (EI) is often set based on a number of factors including scene brightness, and the effective ISO of the digital camera is adjusted to largely match the EI. In determine aperture step 240, an aperture setting 245 is determined to control the f-number of the camera lens in accordance with the measured scene brightness, the capture mode setting 225 and the exposure index setting 235. An exposure time setting 255 (t) is then determined in determine exposure time step 250 in accordance with the scene brightness, the capture mode setting 225, the exposure index setting 235 and the aperture setting 245. It should be noted that these steps are not necessarily performed in the order shown in FIG. 1. After the various settings have been determined, a capture digital image step 260 is used to capture and store a digital image 265. However, the method of the typical camera control system 200 is prone to capture images with poor perceived image quality because the degree of brightness and motion in the scene can be highly variable and since motion is not taken into account, disappointing levels of motion blur or noise can be present in the images.
U.S. Patent Application Publication 2007/0237514 to Pillman et al., entitled “Varying camera self-determination based on subject motion,” teaches a method for capturing digital images where motion in the scene is measured prior to image capture. Various camera settings are adjusted responsive to the determined scene motion. If slow or no scene motion is detected, then additional analysis is done to help select a capture mode setting 225 for the digital camera. If rapid scene motion is detected, then a capture mode setting 225 suitable for sports photography is selected for use by the exposure control system 200 as presented in FIG. 1. The sports capture mode would limit exposure time and use a higher exposure index setting 235 than a typical default capture mode. As such, the method of Pillman primarily provides an improved method for capture of scenes with significant scene motion.
In U.S. Patent Application Publication 2007/0237506 to Minema et al., entitled “Image blurring reduction,” a camera is described wherein an image is captured at a slower shutter speed if no camera motion is detected. If camera motion is detected, then an image is captured at a faster shutter speed. While this method does reduce motion blur in images, it does not address the combined effects of motion blur and noise in the image on the perceived image quality of the image in selecting capture conditions including exposure time and ISO.
U.S. Pat. No. 5,598,237 to McIntyre et al., entitled “Image capture apparatus,” describes an image capture apparatus operable in a hand-held condition and in a stabilized non-hand-held condition. Different exposure parameters are selected depending on whether the camera is being used in the hand-held condition.
U.S. Patent Application Publication 2009/0040364 to Rubner, entitled “Adaptive Exposure Control,” teaches using a multiple image capture process to reduce image quality artifacts including motion blur. FIG. 2 shows a flow chart summarizing this method. In capture first image step 270, a first image is captured using exposure conditions defined by the camera autoexposure control system (e.g., the exposure control system 200 as presented in FIG. 1). In analyze image for deficiencies step 275, the first image is analyzed for aspects of image quality such as overexposure, underexposure, motion blur, dynamic range or depth of field to determine which aspects have been met and where deficiencies remain. Based on this analysis, a remaining deficiencies test 280 is used to check whether any deficiencies remain in the aspects of image quality. If some deficiencies remain, the process proceeds to update exposure parameters step 282 where new exposure parameters are set for at least one additional image. Capture additional image step 272 is then used to capture an additional image using the new exposure parameters. The additional image is then analyzed with the analyze image for deficiencies step 275. This process repeats until the remaining deficiencies test 280 determines that all the aspects of image quality have been met amongst the multiple images that have been captured. A final image is then constructed by combining portions of the multiple captured images using a combine captured images step 285 in such a way that all the aspects of image quality desired are met. However, the method of Rubner does not address motion related image quality issues in applications which require capturing only a single captured image.
U.S. Patent Application Publication 2008/0101786 to Pozniansky et al., entitled “Control of artificial lighting of a scene to reduce effects of motion in the scene on an image being acquired,” describes a method for using artificial illumination to acquire an improved image based on motion analysis. This application teaches use of scene luminance thresholds and a motion blur threshold to determine when flash should be used. This application adds a motion threshold to complement the well known use of a scene luminance threshold in controlling flash. In their FIG. 10, they combine flash and ambient exposure based on thresholds for scene luminance and motion blur. The approach of Pozniansky et al. does not teach a method for continuously blending ambient and flash exposure for preferred exposure control.
U.S. Pat. No. 5,124,740 to Wheeler, entitled “Depth number based technique for selecting lens aperture size and flash parameters for a full flash exposure,” describes a method for controlling a flash and various system parameters to optimize exposure for still image capture. In particular, Wheeler teaches the use of a depth number as well as a guide number to optimize aperture (f-number) selection.
U.S. Pat. No. 5,130,739 to O'Such et al., entitled “Automatic optimization of photographic exposure parameters through determination and utilization of extra system speed,” describes a method for controlling flash and ambient exposure to obtain improved image quality. This patent effectively teaches the adjustment of camera exposure to maximize an overall quality objective, even if that means capturing a scene at an exposure index other than the nominal ISO of film loaded into the camera.
The Wheeler and O'Such et al. patents describe optimization of image capture parameters accounting for many system factors, such as focal length, available flash power, subject distance, ambient illumination, and system speed. However, no explicit motion information is used to improve scene capture, as motion estimation was not taught or used in this art.
As shown by the cited prior art, cameras usually enable flash based only on scene brightness, enabling flash if the scene brightness is below a threshold. Further, present cameras tend to switch over to only flash exposure if the scene brightness is below the threshold. When the subject distance is large relative to the flash power available, a flash-only exposure will also be of poor quality. In such situations, it is preferred to use exposure from both flash and ambient illumination, using an exposure time that allows for significant ambient exposure, not just synchronization with the flash.
In some cases, it is desirable to capture an image with low flash power, either to conserve energy or to capture an image without waiting for the flash capacitor to fully charge. Of the cited prior art, only Wheeler and O'Such address this issue, but neither of them use knowledge of motion.
There remains a need for a method to automatically fully utilize flash (camera-controlled illumination) and ambient illumination when photographing scenes over a wide range of subject distance, scene illumination, and scene velocity.